Dying ReLU

ReLU returns 0 when pre-activation is negative, which is sometimes useful to have simplicity in the network.

But sometimes many neurons have negative values, which zeroes out most of the network activations.

It's solved by:

• Swish (SiLU), by using input in equation connection: $silu(x) = x * \sigma(x)$
• GeLU, by using Gaussian distribution: $gelu(x) = x * \Phi(x)$
• Pairing ReLU with BatchNorm.